{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../../img/ods_stickers.jpg\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 非监督学习应用练习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本次挑战中，我们将对数据降维和聚类方法的工作原理进行练习。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将使用三星提供的 [<i class=\"fa fa-external-link-square\" aria-hidden=\"true\"> Human Activity Recognition 活动识别数据集</i>](https://archive.ics.uci.edu/ml/datasets/Human+Activity+Recognition+Using+Smartphones) 。这些数据来自三星 Galaxy S3 手机的加速度计和陀螺仪，你可以通过上面的链接找到有关这些功能的更多信息）。这些活动类型包括：走路，站立，躺下，坐着或爬楼梯。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来的挑战过程中，我们首先假装不了解活动的类型，并尝试纯粹基于特征对样本进行聚类。然后，我们将确定身体活动类型的问题解决为分类问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们先导入本次挑战可能会用到的模块和函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import warnings\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn import metrics\n",
    "from sklearn.cluster import AgglomerativeClustering, KMeans, SpectralClustering\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import LinearSVC\n",
    "from tqdm import tqdm_notebook\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "%matplotlib inline\n",
    "plt.style.use([\"seaborn-darkgrid\"])\n",
    "plt.rcParams[\"figure.figsize\"] = (12, 9)\n",
    "plt.rcParams[\"font.family\"] = \"DejaVu Sans\"\n",
    "\n",
    "RANDOM_STATE = 17"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，加载并读取数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -nc \"../../data/samsung_HAR.zip\"\n",
    "!unzip -o \"samsung_HAR.zip\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH_TO_SAMSUNG_DATA = \"./samsung_HAR\"\n",
    "X_train = np.loadtxt(os.path.join(PATH_TO_SAMSUNG_DATA, \"samsung_train.txt\"))\n",
    "y_train = np.loadtxt(\n",
    "    os.path.join(PATH_TO_SAMSUNG_DATA, \"samsung_train_labels.txt\")\n",
    ").astype(int)\n",
    "\n",
    "X_test = np.loadtxt(os.path.join(PATH_TO_SAMSUNG_DATA, \"samsung_test.txt\"))\n",
    "y_test = np.loadtxt(\n",
    "    os.path.join(PATH_TO_SAMSUNG_DATA, \"samsung_test_labels.txt\")\n",
    ").astype(int)\n",
    "\n",
    "X_train.shape, X_test.shape, y_train.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "聚类挑战中，我们不需要目标向量。所以，下面合并训练和测试样本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.vstack([X_train, X_test])\n",
    "y = np.hstack([y_train, y_test])\n",
    "\n",
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们尝试查看类别数据中唯一值有哪些，并得到数据集目标类别的数量 `n_classes`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_classes = np.unique(y).size\n",
    "np.unique(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据 [<i class=\"fa fa-external-link-square\" aria-hidden=\"true\"> 数据集说明</i>](https://archive.ics.uci.edu/ml/machine-learning-databases/00240/UCI%20HAR%20Dataset.names)，我们可以得知类别数值表示的含义："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 1 – walking\n",
    "- 2 – walking upstairs\n",
    "- 3 – walking downstairs\n",
    "- 4 – sitting\n",
    "- 5 – standing\n",
    "- 6 – laying down"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，尝试使用 `StandardScaler` 完成特征数据的规范化："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(X)\n",
    "X_scaled.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们将应用主成分分析 PCA 降维来缩减特征的数量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i class=\"fa fa-question-circle\" aria-hidden=\"true\"> 问题：</i>如果使用 PCA 降维，并使得缩减后的数据保留原始数据 90% 的方差，那么得到的特征数量是多少个？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "请使用挑战一开始定义好的 RANDOM_STATE 常量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i class=\"fa fa-question-circle\" aria-hidden=\"true\"> 问题：</i>降维后的第一主成分涵盖了多大比例的方差？舍入到最接近的百分比。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i class=\"fa fa-question-circle\" aria-hidden=\"true\"> 问题：</i>请绘制前两个主成分特征的二维散点图，并使用数据已知类别进行着色。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i class=\"fa fa-question-circle\" aria-hidden=\"true\"> 问题：</i>通过肉眼观察，上图大致分为几个聚类簇？不同簇中又包含哪些类型的活动？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 【A】 1 个簇：6 种类别的活动。\n",
    "- 【B】 2 个簇：`walking, walking upstairs, walking downstairs` 和 `sitting, standing, laying`。\n",
    "- 【C】 3 个簇：`walking` 和 `walking upstairs, walking downstairs` 和 `sitting, standing, laying`。\n",
    "- 【D】 6 个簇。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，请使用 KMeans 聚类方法对 PCA 降维后的数据进行聚类操作。于此同时，这里建议聚集为 6 类，实际上在正常情况下我们一般并不会知道聚为几类。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i class=\"fa fa-question-circle\" aria-hidden=\"true\"> 问题：</i>建立 KMeans 聚类，设置 `n_clusters=n_classes`, `n_init=100` 以及前面提供的 RANDOM_STATE。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i class=\"fa fa-question-circle\" aria-hidden=\"true\"> 问题：</i>同样，请使用 PCA 前面 2 个主成分特征绘制二维图像，但这一次使用 KMeans 聚类标签进行着色。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以对比查看原始标签分布和 KMeans 聚类标签的分布之间的不同之出。接下来，我们分别查看每个原始类别都被 KMeans 聚类划分成了那几个簇。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tab = pd.crosstab(y, cluster_labels, margins=True)\n",
    "tab.index = [\n",
    "    \"walking\",\n",
    "    \"going up the stairs\",\n",
    "    \"going down the stairs\",\n",
    "    \"sitting\",\n",
    "    \"standing\",\n",
    "    \"laying\",\n",
    "    \"all\",\n",
    "]\n",
    "tab.columns = [\"cluster\" + str(i + 1) for i in range(6)] + [\"all\"]\n",
    "tab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "表格行名称是原始标签，而列名称则对应了 KMeans 聚类后的簇序号。可以看到，几乎每一个原始类别都被重新聚为了几个分散簇。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里，我们使用某一原始类别被 KMeans 聚类后的最大数量簇，除以原始类别总数来表征聚类的分散程度。得到的数值越小，即代表聚类后的簇越分散。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table border=\"1\" class=\"dataframe\">\n",
    "  <thead>\n",
    "    <tr style=\"text-align: right;\">\n",
    "      <th></th>\n",
    "      <th>cluster1</th>\n",
    "      <th>cluster2</th>\n",
    "      <th>cluster3</th>\n",
    "      <th>cluster4</th>\n",
    "      <th>cluster5</th>\n",
    "      <th>cluster6</th>\n",
    "      <th>all</th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <th>walking</th>\n",
    "      <td>903</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>78</td>\n",
    "      <td>741</td>\n",
    "      <td>0</td>\n",
    "      <td>1722</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>going up the stairs</th>\n",
    "      <td>1241</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>5</td>\n",
    "      <td>296</td>\n",
    "      <td>2</td>\n",
    "      <td>1544</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>going down the stairs</th>\n",
    "      <td>320</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>196</td>\n",
    "      <td>890</td>\n",
    "      <td>0</td>\n",
    "      <td>1406</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>sitting</th>\n",
    "      <td>1</td>\n",
    "      <td>1235</td>\n",
    "      <td>91</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>450</td>\n",
    "      <td>1777</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>standing</th>\n",
    "      <td>0</td>\n",
    "      <td>1344</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>562</td>\n",
    "      <td>1906</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>laying</th>\n",
    "      <td>5</td>\n",
    "      <td>52</td>\n",
    "      <td>1558</td>\n",
    "      <td>0</td>\n",
    "      <td>0</td>\n",
    "      <td>329</td>\n",
    "      <td>1944</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <th>all</th>\n",
    "      <td>2470</td>\n",
    "      <td>2631</td>\n",
    "      <td>1649</td>\n",
    "      <td>279</td>\n",
    "      <td>1927</td>\n",
    "      <td>1343</td>\n",
    "      <td>10299</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "例如，所示的表格。walking 被重新划分为 cluster1, cluster4, cluster5。其中，数量最多的是 cluster1，则分散程度："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ \\frac{903}{1722} = 0.524$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i class=\"fa fa-question-circle\" aria-hidden=\"true\"> 问题：</i>请计算各原始类别聚类后的分散程度，并按照分散程度由大到小排序。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，请使用实验中介绍的方法，通过求解观测数据点与其所在的簇的质心之间的平方距离之和来选择本次数据的最佳聚类 K 值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i class=\"fa fa-question-circle\" aria-hidden=\"true\"> 问题：</i>选择 KMeans 聚类合适的聚类 K 值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "不出意外时，这里的最佳聚类 K 值应该为 2，也就是将数据聚集为 2 类。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i class=\"fa fa-question-circle\" aria-hidden=\"true\"> 问题：</i>接下来，尝试使用 Agglomerative 聚类，并计算二者（KMeans）的兰德指数 ARI 值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里使用 `n_clusters=n_classes` 时得到的 KMeans 聚类结果即可。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般情况下，ARI 值越高代表聚类效果越好。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: #e6e6e6; margin-bottom: 10px; padding: 1%; border: 1px solid #ccc; border-radius: 6px;text-align: center;\"><a href=\"https://nbviewer.jupyter.org/github/shiyanlou/mlcourse-answers/tree/master/\" title=\"挑战参考答案\"><i class=\"fa fa-file-code-o\" aria-hidden=\"true\"> 查看挑战参考答案</i></a></div>"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
